IMAGE SEQUENCE DESCRIPTION USING SPATIOTEMPORAL FLOW CURVES: TOWARD MOTION-BASED RECOGNITION By

Recovering a hierarchical motion description of a long image sequence is one way to recognize objects and their motions. Intermediate-level and high-level motion analysis, i.e., recognizing a coordinated sequence of events such as walking and throwing, has been formulated previously as a process that follows high-level object recognition. This thesis develops an alternative approach to intermediate-level and high-level motion analysis. It does not depend on complex object descriptions and can therefore be computed prior to object recognition. Toward this end, a new computational framework for low and intermediate-level processing of long sequences of images is presented. Our new computational framework uses spatiotemporal (ST) surface ow and ST ow curves. As contours move, their projections into the image also move. Over time, these projections sweep out ST surfaces. Thus, these surfaces are direct representations of object motion. ST surface ow is de ned as the natural extension of optical ow to ST surfaces. For every point on an ST surface, the instantaneous velocity of that point on the surface is recovered. It is observed that arc length of a rigid contour does not change if that contour is moved in the direction of motion on the ST surface. Motivated by this observation, a function measuring arc length change is de ned. The direction of motion of a contour undergoing motion parallel to the image plane is shown to be perpendicular to the gradient of this function. ST surface ow is then used to recover ST ow curves. ST ow curves are de ned such that the tangent at a point on the curve equals the ST surface ow at that point. ST ow curves are then grouped so that each cluster represents a temporally-coherent structure, i.e., structures that result from an object or surface in the scene undergoing motion. Using these clusters of ST ow curves, separate moving objects in the scene can be hypothesized and occlusion and disocclusion between them can be identi ed. The problem of detecting cyclic motion, while recognized by the psychology community, has received very little attention in the computer vision community. In order to show the representational power of ST ow curves, cyclic motion is detected using ST ow curves without prior recovery of complex object descriptions.